Automated Event Coding Using Conditional Random Fields

We present an approach using conditional random fields (CRFs) for extracting and coding political events from newswire stories. Coding an event from a news story requires the extraction of the actor and target, the event itself, and the date of occurrence. Actors and targets are political entities. Events are classified into 22 discrete categories in the popular WEIS scheme (Tomlinson 1993). Lead sentences of newswire stories are surprisingly complex in structure and we demonstrate the importance of segmenting it into its constituent phrases as a pre-processing step. We design a CRF model that labels each word in a phrase as being part of the actor, target or a specific event type. Using two hundred sentences drawn from Reuters, we compare the performance of our CRF coder against TABARI (Schrodt 2001), an automated event coder in active use in the political science community. Our comparison focuses on two important WEIS event categories (force (22) and comment (02)). We demonstrate that on the difficult to code force category our CRF coder performs with an accuracy of 72%, recall of 70% and precision of 91%. In contrast, TABARI performs with an accuracy of 22%, recall of 7% and precision of 50%. We explain the sources of power in the CRF model and conclude by describing extensions to our model to code events in all 22 WEIS categories.